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DETAILED ACTION 
Specification 

1 . The amendments to the specification overcome the objections made in the 
previous Office Action. The objections to the specification are withdrawn. 

Response to Amendment 

2. The amendments to the claims have been entered. Claims 1-6, 10, 13, 14, 16, 
1 9-21 , 24, 28, 32-34, and 38-42 are currently amended and new claims 46-55 have 
been added. 

Response to Arguments 

3. Applicant's arguments with respect to claims 1-45 have been considered but are 
moot in view of the new ground(s) of rejection. 



Claim Rejections - 35 USC § 112 

4. The following is a quotation of the first paragraph of 35 U.S.C. 112: 

The specification shall contain a written description of the invention, and of the manner and process of 
making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the 
art to which it pertains, or with which it is most nearly connected, to make and use the same and shall 
set forth the best mode contemplated by the inventor of carrying out his invention. 

5. Claims 16 is rejected under 35 U.S.C. 112, first paragraph, as failing to comply 
with the written description requirement. The claim(s) contains subject matter which 
was not described in the specification in such a way as to reasonably convey to one 
skilled in the relevant art that the inventor(s), at the time the application was filed, had 
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possession of the claimed invention. There is no indication in the specification of 
specifically notifying the human who is to perform a subtask the maximum amount of 
time to be spent on that subtask. Page 13, lines 16-23 describe that the Task Server 
102 describes to the Junta server 106 a maximum time attribute for a subtask, but there 
is no description of the Junta server 106 passing this information to a human user at a 
Junta node. 



Claim Rejections - 35 USC § 102 

6. The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that 
form the basis for the rejections under this section made in this Office action: 

A person shall be entitled to a patent unless - 

(e) the invention was described in (1 ) an application for patent, published under section 122(b), by 
another filed in the United States before the invention by the applicant for patent or (2) a patent 
granted on an application for patent by another filed in the United States before the invention by the 
applicant for patent, except that an international application filed under the treaty defined in section 
351(a) shall have the effects for purposes of this subsection of ah application filed in the United States 
only if the international application designated the United States and was published under Article 21(2) 
of such treaty in the English language. 

7. Claims 1 -4, 6-1 5, 1 9-22, 24-30, 32-35, 37-46, 49, 50, and 53-55 are rejected 
under 35 U.S.C. 102(e) as being anticipated by Bejar et al. (U.S. Patent 6,295,439). 

In regard to claims 1,19, and 32, Bejar et al. disclose a method, storage 
medium, and apparatus comprising a storage medium for a computer system to user 
human assistance in performing tasks, the method comprising: 

automatically and under control of a first computer system (Fig. 1,10, column 4, 



lines 50-53), causing a task (assessment) to be performed by, 
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identifying a first and second subtask of the task (the task of evaluating scores 
awarded by human evaluators to constructed responses is broken into subtasks by 
assigning each constructed response to a category, each category representing a 
subtask to be completed, Fig. 1 1A, step 303, column 17, lines 63-67); 

identifying one or more required capabilities of a human for performance of the 
first subtask (each category is an area of expertise to which the human evaluator is 
assigned, step 305, column 17, line 67 to column 18, line 2 and column 8, lines 7-10 
and lines 15-24); 

dispatching the first subtask to a remote second computer system of a first 
human for performance by said first human, the first human identified as being one or 
more humans who have the capabilities that satisfy the required capabilities of the first 
subtask (step 31 1 , the constructed responses to be evaluated are transmitted to the 
human evaluators at assessment stations, column 18, lines 14-19 and column 5, lines 
24-32); 

receiving a first result from said first human via the second computer system, the 
first result generated by performance of said subtask by said first human (Fig. 1 1 B, step 
315, the constructed responses are scored by the human evaluator and the scores are 
transmitted back from the assessment stations, column 18, lines 30-32); and 

generating a result for said task based at least in part on said first result (step 
321 , the scores awarded by the human evaluator are analyzed, column 18, lines 47-51). 
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In regard to claims 2, 20, and 33, Bejar et al. disclose the identifying by the first 
computer system of the first and second subtasks of the task includes decomposing the 
task into at least the first and second subtasks (the task of evaluating a plurality of 
constructed responses is broken down by assigning the constructed responses to 
categories, column 17, lines 52-67). 

In regard to claims 3, 21 , and 34, Bejar et al. disclose the method further 
comprises the first computer system dispatching said second subtask to a third 
computer system of a second human for performance by the second human (step 307, 
a second human evaluator is assigned to at least one of the same categories as the first 
human evaluator, column 18, lines 2-5), and the first computer system receiving a 
second result from the second human via the third computer system for the second 
subtask (at step 315 when scores from all the evaluators are transmitted, column 18, 
lines 30-32 and lines 35-37); and 

the first computer system further bases its generation of the result for said task 
on said second result (scores of all the evaluators are analyzed, column 18, lines 47- 
51). 

In regard to claims 4, 22, and 35, Bejar et al. disclose said task further 
comprising a third subtask, and the method further comprises the first computer system 
receiving and performing said third subtask producing a third result; and 



Application/Control Number: 09/976,717 Page 6 

Art Unit: 2655 

the first computer system further bases its generation of the result for said task 
on said third result (Fig. 2B, step 107, a subtask of the overall task of evaluating scores 
awarded by human evaluators to constructed responses includes a subtask of 
determining the correct application to launch for the evaluator, column 7, lines 18-25, 
this subtask is necessary to launch the correct application for the human evaluator, thus 
the generation of the result for the overall task is 'based on' this result, column 8, lines 
51-57). 

In regard to claims 6, 24, and 37, Bejar et al. discloses said first human is one of 
college educated, at most high school educated, at most elementary school educated, 
and not formally educated (every human is one of these education levels; that is, these 
four education levels represent every level of education a human could possibly have, 
therefore, any human will be one of college educated, at most high school educated, at 
most elementary school educated, and not formally educated). 

In regard to claims 7, 25, and 38, Bejar et al. discloses said subtask is one of 
text, speech, sound, and images related operations (the constructed responses 
evaluated by the human evaluators comprise text documents, audio responses, and 
drawings, column 9, lines 41-50 and column 11, lines 3-5). 

In regard to claims 8, 26, and 39, Bejar et al. discloses the result is one of text, 
numbers, tuples, and sound (column 11, lines. 5-7). 
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In regard to claims 9, 27, and 40, Bejar et al. discloses said task is one of text 
classification, image comparison, image processing, speech comparison, speech 
recognition, conversion of speech into text, and comparison of music samples (the 
human evaluator classifies the constructed responses by assigned each constructed 
response a score, column 18, lines 23-57). 

In regard to claims 10, 28, and 41 , Bejar et al. discloses said task is associated 
with multiple attributes related to performance of said task, the attributes including an 
accuracy attribute, a security attribute, a timeout attribute, a maximum time spent 
attribute, a maximum cost per task attribute, and a maximum total cost attribute, and 
wherein the identifying of the one or more humans, the dispatching of the first subtask, 
and the generating of the result for said task are performed in a manner to reflect the 
multiple associated attributes (column 10, lines 3-10 and column 13, lines 22-36). 

In regard to claim 1 1 Bejar et al. discloses said task is associated with one or 
more attributes, and said attribute includes an accuracy attribute (column 13, lines 31- 
36). 

In regard to claims 12, 29, and 42, Bejar et al. discloses dispatching said first 
subtask to N-1 additional humans to perform said subtask and said accuracy comprises 
a selection of one of majority governs, and at least N2 agreed answers, wherein N2 and 
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N1 are integers, with N2 greater than N1 (column 15, lines 24-28 and column 16, lines 
31-37). 

In regard to claims 13 and 43, Bejar et al. disclose tracking the accuracy of 
additional humans (the accuracy of all evaluators is tracked, column 13, lines 31-36). 

In regard to claims 14 and 44, Bejar et al. disclose said generation of the result 
further takes into consideration the accuracy of the additional humans (column 19, lines 
23-26). 

In regard to claims 15, 30, and 45, Bejar et al. discloses said task is associated 
with one or more attributes including a security attribute, and said security attribute 
comprises a selection of one of a strict security level, a lax security level, and no 
security level (constructed responses are executed in write protected mode or view only 
mode, column 9, lines 31-41). 

In regard to claim 46, Bejar et al. disclose the first computer system is a task 
server system that is part of a distributed hybrid computer/human computation 
arrangement, and wherein the first human is one of numerous human remote from the 
task seer system who each use distinct client computing devices to act as nodes of the 
distributed hybrid computer/human computation system (Fig. 1, column 5, lines 24-32). 
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In regard to claim 49, Bejar et al. disclose the dispatching by the first computer 
system of the first subtask to the remote second computer system is performed using a 
defined application programming interface ("API") (see appendix A). 

In regard to claim 50, Bejar et al. disclose the dispatching by the first computer 
system of the first subtask to the remote second computer system is performed by 
programmatically sending one or more messages from the first computer system to the 
remote computer system (column 5, lines 28-32). 

In regard to claim 53, Bejar et al. disclose the required capabilities of the human 
for performance of the first subtask include an ability to speak a specified language 
(English, column 8, lines 26-29). 

In regard to claim 54, Bejar et al. disclose the required capabilities of the human 
for performance of the first subtask include an ability to hear (constructed response may 
be audio, column 11, lines 3-5). 

In regard to claim 55, Bejar et al. disclose the required capabilities of the human 
for performance of the first subtask include a specified degree of historical accuracy by 
the human when performing subtasks (column 13, lines 31-36). 
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Claim Rejections - 35 USC § 103 

8. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 102 of this title, if the differences between the subject matter sought to be patented and 
the prior art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 

9. Claims 5, 23, and 36 rejected under 35 U.S.C. 103(a) as being unpatentable over 
Bejar et al., in view of Burstein et al. (U.S. Patent 6,181,909). 

Bejar et al. does not disclose the first computer system additionally performs said 
second subtask (evaluates a constructed response) and the first computer system 
further bases its generation of the result for said task on said second result. 

Burstein et al. disclose a method and system for performing a subtask (scoring 
an essay) to produce a result (essay score, column 3, lines 45-51 ). 

It would have been obvious to one of ordinary skill in the art at the time of 
invention to modify Bejar et al. to perform said second subtask (scoring of one of the 
categories of constructed responses) at the first computer using the method of 
automatic essay scoring disclosed by Burstein et al., in order to provide a standard 
score by which the human evaluators could be assessed. 

10. Claims 17, 18, 31, 51, and 52 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Bejar et al., in view of Paiziz (U.S. Patent 6,338,042). 
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In regard to claims 17 and 18, Bejar et al. do not disclose said task is associated 
with one or more attributes, and said attributes include a maximum cost per task 
attribute or a maximum total task cost attribute. . 

Paiziz discloses a method wherein a task (contribution) is associated with one or 
more attributes, and said attributes include a maximum cost per task (Fig. 4, 418, each 
individuals contributions are given a maximum market reference point, column 8, lines 
36-40) and a total task cost attribute (414, total overall budget available to pay the 
individuals in the given role, column 8, lines 23-25). 

It would have been obvious to one of ordinary skill in the art at the time of 
invention to modify Bejar et al. to associate a maximum cost per task attribute or a 
maximum total task cost attribute with the task, so that monetary compensation 
decisions for individuals could be automatically determined according to the individuals 
contributions, as taught by Paiziz, column 2, lines 47-51). 

In regard to claim 31 , Bejar et al. disclose said task is associated with one or 
more attributes, wherein the attributes include a maximum time to be spent on a task 
(elapsed time to award a score, column 10, lines 3-7). 

Bejar et al. do not disclose said attributes include a maximum cost per task 
attribute or a maximum total task cost attribute. 

Paiziz discloses a method wherein a task (contribution) is associated with one or 
more attributes, and said attributes include a maximum cost per task (Fig. 4, 418, each 
individuals contributions are given a maximum market reference point, column 8, lines 
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36-40) and a total task cost attribute (414, total overall budget available to pay the 
individuals in the given role, column 8, lines 23-25). 

It would have been obvious to one of ordinary skill in the art at the time of 
invention to modify Bejar et al. to associate a maximum cost per task attribute or a 
maximum total task cost attribute with the task, so that monetary compensation 
decisions for individuals could be automatically determined according to the individuals 
contributions, as taught by Paiziz, column 2, lines 47-51 ). 

In regard to claim 51 , Bejar et al. do not disclose providing information to the first 
human of a payment associated with performance of the first subtask if the human 
chooses to perform the subtask. 

Paiziz discloses a method in which a payment associated with the performance 
of a subtask (contribution) if the human chooses to perform the subtask (suggested pay 
according to an individual's contribution, Fig. 4, step 360, column 7, lines 41-45). 

It would have been obvious to one of ordinary skill in the art at the time of 
invention to modify Bejar et al. to provide information to the first human of a payment 
associated with a subtask, so that the first human could determine whether the pay was 
acceptable, as taught by Paiziz (column 7, lines 41-45). 

In regard to claim 52, Bejar et al. do not disclose providing to the first human a 
payment associated with the first subtask. 
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Paiziz disclose a method in which a human is provided a payment associated 
with a subtask (contribution, Fig. 4, step 364 a payment decision is made, column 7, 
lines 45-54). 

It would have been obvious to one of ordinary skill in the art at the time of 
invention to modify Bejar et al. to provide to the first human a payment associated with 
the subtask, so the human could be appropriately compensated for services rendered. 

1 1 . Claims 47 and 48 are rejected under 35 U.S.C. 1 03(a) as being unpatentable 
over Bejar et al., in view of Christensen et al. (U.S. Patent 5,881 ,230). 

In regard to claim 47, Bejar et al. discloses that the services of the task server 
system and the coordinating server (i.e. coordinating the performance of the first 
subtask by identifying the first human as having capabilities that satisfy the identified 
required capabilities and sending to the second computer system of the first human an 
indication of the first subtask to be performed) are performed on the same physical 
server (Fig. 1, assessment system 10, column 17, lines 63-67; column 17, line 67 to 
column 18, line 2; column 8, lines 7-10 and lines 15-24; column 18, lines 14-19; and 
column 5, lines 24-32). 

Bejar et al. do not disclose that the coordination services are performed on a 
distinct coordination server remote from the task server system, wherein the task server 
system sends information indicating the first subtask and required capabilities to the 
remote coordination server. 
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Christensen et al. disclose a method for implementing application objects on 
remote computers as needs dictate so that a given logical model can be implemented 
independently of the underlying physical model (column 13, line 62 to column 14, line 
27). 

It would have been obvious to one of ordinary skill in the art at the time of 
invention to modify Bejar et at. so the services of the task server and the services of the 
coordinating server were implemented on separate physical servers with the 
appropriate information sent between two, so that the physical performance and 
administration needs of the system could be addressed without giving up the logical 
model, as taught by Christensen et al. (column 13, lines 62-67). 

In regard to claim 48, Bejar et al. disclose the coordinating services identify a first 
and second subtask of the task (column 17, lines 63-67), wherein the coordinating 
services causes the result to be generated for the task based at least in part on the first 
result (column 18, lines 30-32), wherein the human is one of numerous humans who 
each use distinct client devices to act as nodes available to the coordinating server 
computing system (column 5, lines 24-28). 

Bejar et al. do not disclose that the coordination services are performed on a 
distinct coordination server remote from the task server system, wherein the task server 
system sends information indicating the first subtask and required capabilities to the 
remote coordination server. 
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Christensen et al. disclose a method for implementing application objects on 
remote computers as needs dictate so that a given logical model can be implemented 
independently of the underlying physical model (column 13, line 62 to column 14, line 
27). 

It would have been obvious to one of ordinary skill in the art at the time of 
invention to modify Bejar et al. so the services of the task server and the services of the 
coordinating server were implemented on separate physical servers with the 
appropriate information sent between two, so that the physical performance and 
administration needs of the system could be addressed without giving up the logical 
model, as taught by Christensen et al. (column 13, lines 62-67). 

Conclusion 

12. The prior art made of record and not relied upon is considered pertinent to 
applicant's disclosure. Knowles et al. (U.S. Patent 6,751,351) disclose a method for 
human individuals to perform character recognition at remote terminals. 

1 3. Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Brian L. Albertalli whose telephone number is (571) 272- 
7616. The examiner can normally be reached on Mon - Fri, 8:00 AM - 5:30 PM, every 
second Fri off. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Wayne Young can be reached on (571) 272-7582. The fax phone number 
for the organization where this application or proceeding is assigned is 571-273-8300. 
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Information regarding the status of an application may be obtained from the 
Patent Application Information Retrieval (PAIR) system. Status information for 
published applications may be obtained from either Private PAIR or Public PAIR. 
Status information for unpublished applications is available through Private PAIR only. 
For more information about the PAIR system, see http://pair-direct.uspto.gov. Should 
you have questions on access to the Private PAIR system, contact the Electronic 
Business Center (EBC) at 866-217-9197 (toll-free). 



BLA 9/28/05 



W.R.Y0ljNG 
PRIMARY EXAMINER 




